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1. Introduction 


This study provides a detailed cost analysis of an education reform implemented throughout the state of 
Kentucky called the Targeted Interventions (TI) program. The following chapter includes an introduction 
of the policy context and state requirements for the program implemented in Kentucky. In addition, it 
outlines the goals of the study, corresponding research questions, and analyses conducted to address 
these questions. 


1.1 Context and Description of the Targeted Interventions Program 


In 2009, the Kentucky legislature signed Senate Bill 1 (2009 RS SB 1) into law, which called for a 
complete overhaul of Kentucky’s assessment and accountability system for PreK—12 education. Since 
then, the Commonwealth has adopted a comprehensive system of education reform—known as 
Unbridled Learning—to monitor, document, and support progress toward the goals of college and 
career readiness for all students. Kentucky’s Next Generation Learners program is one of the strategic 
priorities established by the Kentucky Board of Education in response to Senate Bill 1. In support of this 
strategic priority, the Kentucky Department of Education (KDE) designed and implemented the Targeted 
Interventions (Tl) program. 


The TI program is integrated into the Kentucky Systems of Interventions framework, with the goal of 
intervening early in a student’s educational career to ensure students leave high school ready to pursue 
opportunities in higher education and the workforce. This program, in effect since the 2010-11 school 
year, is designed to monitor student progress on college- and career-readiness proficiency standards, 
screen students who need support, and provide interventions that will address those needs. Specifically, 
KDE defines targeted interventions as “planned, carefully considered interventions that occur when 
students do not meet the grade level expectations (i.e., benchmarks) that are necessary for academic 
progress” (KDE, 2018). The TI program targets Grades 8, 10, and 11. Each intervention cycle typically 
consists of a diagnostic pretest, the development of instructional targets and associated formative 
assessments, direct delivery of instruction, and a post-test or other measurement to demonstrate the 
expected milestone had been achieved. Supports for students who do not meet statewide benchmarks 
are required by law to be offered in order to enable students to enter college without the need for 
developmental or remedial coursework (KDE, 2019a). These supports are sometimes referred to as 
transitional interventions. Every district has the autonomy to determine how best to implement the 
intervention process to meet the needs of their students. Interventions are delivered in a variety of 
fashions, including as full courses (e.g., within a school’s regular schedule as existing courses or stand- 
alone courses) and/or outside of the regular school day (e.g., offered as before-/after-school activities). 


While on-the-ground implementation details are determined by districts, the Unified Strategic Plan 
defines a uniform process for identifying students in need of intervention services (KDE, 2019b). All 
Kentucky students in Grades 8, 10, and 11 were supposed to take grade level-appropriate tests within 
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the ACT® suite of assessments,' and those who did not meet statewide benchmarks’ were supposed to 
receive a referral to the Tl program. As mentioned above, all intervention programs are intended to 
focus on preparing students to demonstrate college and career readiness by the end of high school (KRS 
158.6453, 2017; KRS 158.6459, 2017; 704 KAR3:305, 2018). 


1.2 Study Purpose and Overview 


The American Institutes for Research (AIR) conducted two separate but related studies to understand the 
implementation of the TI program. The first study analyzed the effects of the program on various college 
outcomes such as remedial course taking in college, enrollment and pass rates of introductory college- 
level courses, student persistence, and credit accumulation (Xu, Backes, Oliveira, & Goldhaber, 2020). This 
study gives policymakers important information about how a statewide high school remediation initiative 
aimed at college and career readiness affects student achievement. As a companion analysis, the current 
work focuses on identifying the costs associated with the implementation of the TI program during the 
2014-15 and 2015-16 school years.’ The cost information reported here is intended to inform 
practitioners throughout Kentucky who are implementing interventions to promote college and career 
readiness. However, with high school remediation programs similar to Kentucky’s Tl program becoming 
increasingly popular,* stakeholders in other states considering similar education policies may also find this 
information helpful as they navigate implementation of similar interventions. In sum, the intention of this 
research is to provide information vital to understanding the costs of the Kentucky TI program and to add 
to the greater body of literature on the costs associated with interventions intended to promote college 
and career readiness. 


To provide stakeholders and policymakers with high-quality cost information about this statewide 
policy, this study answers the following research questions: 


1. What is the overall cost per student of providing services under the Tl program? 


2. What are the costs per student of the Tl program associated with different types of 
personnel and non-personnel resources? 


3. What are the costs per student associated with the common implementation activities 
performed under the Tl program? 


This work uses the ingredients approach to cost analysis (Levin et al., 2018) and a resource cost model 
(RCM) to organize data on the quantities and prices of resources used to deliver program services. 
Together, the research approach and data organization were used to produce several important, 


1 For more information about the ACT® suite of assessments, visit https://www.act.org/content/act/en/k12-educators-and- 
administrators.html. 

2 The Unified Plan defines benchmark scores for 11th-grade students as “ACT scores of 18 for English, a score of 20 for Reading, 
and a Mathematics score of 19 for some introductory courses in Mathematics (often statistics or an applied Mathematics 
course), a 22 for college algebra, and a 27 for calculus ... SAT equivalent scores may also be used” (p. 7). 

3 Note that the current study is limited to interventions received by targeted 11th-grade students in the 2014-15 and 2015-16 
school years. This is because individual student-level intervention data collection started in 2014-15, and reporting was 
mandatory for 11th-grade students only. 

4 The number of statewide high school remediation programs doubled between 2013 and 2017. Currently, there are 17 
statewide high school remediation programs, and 22 states offer local high school remediation programs (Barnett, Chavarin, & 
Griffin, 2018). 
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contextualized cost estimates for the TI program. A diverse sample of schools was selected to 
investigate the resources involved in the implementation of the Tl program across the state. Data 
collection focused on identifying all resources used to implement the TI program at each site. The data 
collected included information about the design of the program and how services were provided to 
students. Qualitative data from across the study sites were also captured to provide a comprehensive 
narrative describing implementation and the associated cost estimates. 


The study results provide the overall per-student cost for the entirety of the Tl program for each study site 
along with the average across the sites. The per-student costs are further broken out by the types of 
resources used to implement the program (i.e., personnel, materials/equipment, facilities, and training). We 
then provide a breakdown of the per-student costs across a set of activity types (administration, instruction, 
monitoring progress, professional development, and student referrals). Finally, we attempt to make the 
results more practical by considering the costs of the TI program in the context of overall, current per- 
student spending at the study sites reported by KDE. For each cost analysis provided in this study, costs are 
presented in a per-student fashion for each site and as an average cost across all study sites. 


The remainder of the study is organized as follows: Section 2 provides an overview of the methods and 
data used. Section 3 includes the main cost results. Section 4 presents a discussion on ways in which the 
findings might be used to inform policy and future research activities. 
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2. Methods and Data 


A rigorous approach to cost analysis requires that the data collected provide a comprehensive account 
of all personnel and non-personnel resources used and their corresponding costs. The following chapter 
describes the methods used to determine the quantities and types of resources utilized to implement 
the Tl program, their prices, and subsequent costs. The data collection process and study sample are 
also explained in detail. This includes information about how data were organized and analyzed to 
determine the per-student costs of the TI program across categories of resource and activity, and for the 
entire program. 


2.1 Methodology 


2.1.1 The Ingredients Approach 


Analysis of the costs associated with complex, multifaceted programs requires a specialized method that 
accounts for all resources involved in the implementation of the program. Basing cost analysis on 
expenditure and budget documents most often fails to convey the true costs associated with a program 
(Levin & Belfield, 2015). Therefore, the cost analysis for this study employs the ingredients method 
(Levin et al., 2018) to identify all resources and portions of resources used in the TI program. The 
ingredients approach is a rigorous cost analysis method that enables researchers to isolate the costs 
associated with individual programs by identifying and assigning prices to all personnel and non- 
personnel resources (ingredients) involved in the program, regardless of their source (e.g., existing 
resources that have been redirected for use by the program, volunteered effort, etc.). In this way, the 
ingredients approach finds its roots in calculating the economic opportunity cost associated with all 
resources used to deliver an intervention. The study team took three general steps to apply the 
approach: identify ingredients, assign prices, and calculate costs. 


Identifying ingredients involved in the program required in-depth investigation that involved reviews of 
documentation and extant data, as well as primary data collection through interviews and surveys. By 
identifying the ingredients used in a program, we achieved a detailed understanding of the program’s 
components and their respective specifications. The Tl program provides flexibility in how schools 
implement the program and deliver services, which made it particularly important to understand 
implementation details at the school level. Information collected at the school level provided fine- 
grained detail about how resources were leveraged to implement the program. In addition, tracking 
implementation at the school level gives the state a better understanding of how the program flexibility 
offered was leveraged by sites to meet students’ needs. 


Key to this approach is the specification of each resource used to deliver program services. It was 
essential that during data collection (described in detail below) the types and quantities of each 
resource were meticulously recorded. The resource specifications for this study included information 
about facilities, personnel, materials, equipment, and software that were essential to the 
implementation of the program. 


The next step in the ingredients approach is to assign prices to the resources. The ingredients approach 
focuses on identifying the monetary value of all resources used for the program of interest. This is done 
by applying corresponding prices to each of the resources identified and using resource quantities to 
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calculate costs. Simply put, calculating costs involves multiplying the quantity of each resource by its 
corresponding unit price. Importantly, by assigning dollar values to every resource input, the ingredients 
method captures the opportunity cost involved in implementing the program (i.e., the dollar value of 
those resources dedicated to the program instead of some other purpose), regardless of the source 
from which the resources were provided. 


Producing accurate cost estimates requires the use of a highly structured system of organizing the cost 
data described above. Our cost estimate calculations relied heavily on the development of an RCM both 
to facilitate the collection of resource specifications (i.e., quantities, prices, etc.) and to calculate prices. 
The following subsection describes the RCM and how it was tailored for use in this cost analysis. 


2.1.2 The RCM 


An RCM is a structured system used to support data collection and analyze cost analysis results. The 
RCM shares its origin with the ingredients approach to cost analysis. This system enables researchers to 
accurately calculate the cost of programs using specific implementation information and to compile 
pricing data. For the current study, site-specific RCMs were used to organize and analyze resource 
specification information within and across all sites. The RCM supported the overall cost calculations 
and was critical in the calculation of more fine-grained cost information. 


In an RCM, every program resource is assigned to a specific resource type category. The resource types 
used for this study are listed in Exhibit 1. Researchers can use the resource categories as a guide to ensure 
that all resources are being considered, not just those that are described in a budget, policy, or program 
manual. Furthermore, the separation of resources into categories helps avoid double counting of similar 
resources. Finally, cost estimates of categorized resources provide important information to potential 
adopters of the policy in terms of laying out the resource requirements for implementing the program. 


Exhibit 1. Intervention Program Resources Types Captured Across Study Sites 


Resource Type Definition 


Personnel Includes the time school- and district-level staff dedicate to the program (for 
example, time spent in meetings regarding the intervention, referring students to 
interventions, delivering instruction, monitoring student progress, grading 
student work, etc.). 


Materials and Includes materials and equipment used in the operation and implementation of 

Equipment the program, such as textbooks, software, computers, interactive boards, and 
other required supplies.” 

Facilities Includes all costs associated with using physical facilities to implement the 


program, primarily capturing the spaces that were used for direct delivery of 
intervention services (classrooms) and meetings (office space). 


Tuition Includes the time and cost of attending conferences and professional 
development sessions related to the Tl program. 


In the process of collecting data, it became clear that there were similar program activities that occurred 
repeatedly across the study sites. For example, every site had administrative duties associated with the 


5 Note that the cost of the ACT suite of assessments is not included in this work, as the assessment is used for purposes other 
than solely to identify students in need of TI program services. 
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implementation of the Tl program that included facilitating meetings, developing interventions, and 
managing logistics within schools. Another example is the body of activities associated with the TI 
program referral process, such as reviewing student ACT scores and assigning them to appropriate 
interventions. The cost study team synthesized information on the various tasks involved in program 
implementation to develop the activities types listed in Exhibit 2. These common activities were 
included in the RCM and analyzed further to understand their associated costs. Reporting costs by 
activity type may help future adopters understand how this policy fits into existing activity and 
procedural structures. 


Exhibit 2. Intervention Program Activities Captured Across Study Sites 


Activity Type | Definition 

Administrative Administrative tasks and duties related to the Tl program, including designing 
and modifying how the program is implemented at schools 

Instruction Delivery of TI program instruction to students including direct instruction, 
grading, preparing lessons, software, and materials 

Monitoring Activities related to supporting and monitoring Tl program implementation 

Professional Organizing and participating in professional development opportunities related 

Development to the Tl program, including training on software used to deliver interventions 

Referral Assessing, referring, and enrolling students into the Tl program, as well as 
tracking their attendance and completion 


Resource specification information collected at each site was entered into site-specific RCMs along with 
relevant pricing information. This information was used to calculate the per-student cost of the Tl 
program overall, as well as by resource type and program activity for each of the study schools. 


2.2 Data Collection 


The AIR cost study team began the data collection by reviewing documents to better understand the TI 
program and its requirements. The study team also had multiple informal conversations with 
practitioners and experts at KDE and individual schools in the state. During these conversations, it 
became clear that schools implemented unique interventions under the TI program. Therefore, the data 
collection had to be designed to capture data from a variety of implementation structures. The study 
team took a multipronged approach to gathering data. Surveys were sent to certificated staff members 
involved in the Tl program, and interviews were conducted with both school- and district-level staff with 
exceptional knowledge of the program’s implementation. 


2.2.1 Documentation Review and Framework Development 

Extant materials from KDE regarding the Tl program were used to understand the structure of the program 
and the requirements of schools and districts. Where details were unclear during this initial phase, the 
study team conducted telephone discussions with selected state, district, and school staff knowledgeable 
about the program to fill these gaps. Using this information, the study team began developing survey and 
interview protocols that would capture all the essential information about resources used in the 
implementation of the Tl program across the study sites. This information also helped frame the 
development of the RCM template used to organize the data collected from the study sites. 
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2.2.2 Instrument Development and Deployment 


It is important to recognize that the data collection for this study was retrospective in nature. The TI 
program analyzed in the current study was implemented in the 2014-15 and 2015-16 school years, yet 
data were collected in 2019. Given the 4-year gap between implementation and data collection, it was 
determined that collecting multiple types of data and looking for overlapping information across the 
reported data sources would lead to more accurate results. A combination of staff surveys and 
interviews served as the primary data sources for this work. This combination contributed to a detailed 
and contextualized account of the resources that were used to implement the TI program at the various 
study schools. The use of surveys not only enabled participants to take their time answering detailed 
questions about time previously spent on TI program activities but also enabled the study team to 
collect data from a larger number of staff members than could be done solely with interviews. 
Interviews at the school and district levels gave more specific and contextual information about the 
implementation of the TI program. 


Surveys were delivered online and designed in a way that respondents would be routed to the 
appropriate set of questions based on their answers to preliminary screening questions (i.e., skip logic). 
The survey asked about the individual’s involvement in the delivery of intervention services, time spent 
preparing for those services, and any related professional development.® Survey data on resource 
quantities were then compiled and added to the site-specific RCMs. 


District- and school-level interview protocols were developed, which included topics that were not as 
easily captured in the online survey. For example, the interview protocols included questions about the 
intervention referral process, the plan used to communicate the TI program to staff, design of the 
interventions offered, and administrative components of the program. The semistructured interview 
protocol gave researchers the flexibility to order questions in a way that accommodated the natural 
progression of the individual conversations.’ The interview audio was recorded with permission from 
the subject for reference purposes. Once the interviews were completed, two researchers from the 
study team independently coded the information, reconciled their work for accuracy, and entered the 
relevant resource information into the RCM. Additional information included in the interview responses 
that added to the context of the program was noted for later discussion. 


2.2.3 Study Sample 


To represent the diversity of contexts in which the TI program was implemented, a stratified sample was 
drawn from all traditional public high schools in the state. Mainstream education finance literature has 
long recognized that both the degree of urbanicity and student family income are key drivers of the cost 
of providing educational services (Duncombe, Hoang, & Yinger, 2015). In addition, research on 
educational achievement shows that family income is a major predictor of significant variation in 
student test scores (Reardon, 2011). Therefore, the sample of school sites in this work was drawn from 
strata based on urbanicity and the percentage of 11th-grade students who qualified for free or reduced- 
price lunch (FRPL) in the second study year (i.e., the 2015-16 school year).® Specifically, the population 


6 See Appendix A for a copy of the staff survey. 

7 See Appendix B for a copy of the protocol used for the district and school staff interviews. 

8 In addition to the dimensions recommended by the education finance literature, we could have also stratified the sample of 
schools by intervention type. Unfortunately, there were no prescribed intervention models that schools and districts could 
choose from a priori that we could draw upon for sampling purposes. 
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of high schools in the state was categorized by district locale (urban, suburban, small town, rural 
remote) and by low, medium, and high FRPL according to tercile in the statewide distribution.’ Prior to 
sample selection, schools and districts that demonstrated low Tl implementation compliance were 
removed from the pool.?° 


Exhibit 3 displays the number of schools statewide within each stratum in school year 2015-16. The 
study team randomly selected two schools from each stratum without replacement. Specifically, we 
performed an initial first pick where one school was sampled from each stratum and then a second-pick 
sample without replacing those that were sampled as first picks. The first-pick set of schools constituted 
the primary sample, and the second-pick schools served as the backup sample if participation was not 
secured from schools in the primary sample." 


Exhibit 3. Numbers of Kentucky Public High Schools by Locale and Free/Reduced Price Lunch 
Tercile in 2015-16 


Suburb 30 9 18 57 
Town 25 29 11 65 
Rural 17 36 47 100 
Total 78 77 77 232 


After the sample was drawn, the districts in which each selected school resided were sent an overview 
of the study and contacted to discuss what participation in the study would entail.1? With the district’s 
permission, individual schools were then sent the study overview and a formal request to participate. In 
cases in which both the school and district agreed, the study team attempted to conduct two 
interviews—one at the district level and one at the school level—and administer an online survey to all 
certificated staff at the school. If a district or school did not consent to participation, they were dropped 
from the sample, and the corresponding backup school and respective district were contacted. 


Not all contacted districts and schools agreed to participate in the study. Additionally, some schools and 
districts that initially agreed to participate did not submit any completed surveys or submit to 
interviews. Data were successfully gathered from a total of eight schools within seven districts. This 
included six district-level interviews, seven school-level interviews, and survey responses from staff at 


° Urbanicity was measured using the district locale code contained in the National Center for Education Statistics Common Core 
of Data (available at https://nces.ed.gov/ccd/ccddata.asp), and school-level incidence of students from low-income families 
(i.e., those eligible for the National School Lunch Program) was derived from the same data source. The cut-points used to 
define the statewide terciles of school-level FRPL percentage were 51.6% and 63.5%. 

10 To determine compliance rates, the AIR impact study team checked the discontinuity of the TI program participation rate 
around the benchmark threshold scores. When there was no discontinuity around the cutoff, the district was deemed 
noncompliant. For more details see Xu, Backes, Oliveira, & Goldhaber, 2020. 

11 The reader will note that one stratum included only one school (schools located in City districts belonging to the 3" tercile of 
poverty). In this case, the one school was selected as a first pick, while the second-pick backup was selected from the stratum 
defined by schools located in Urban districts belonging to the 2" FRPL tercile. This was done to preserve representation of 
Urban locales in the sample. It was the case that the first pick was not able to participate, which required us to draw upon the 
second pick from neighboring Urban/2"¢ FRPL tercile strata. 

12 See Appendix C for the study overview provided to districts and schools. 
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five schools. After data collection was completed, two schools were removed from the analysis due to 
insufficient data, leaving data from six schools upon which the analysis results are based. Exhibit 4 
shows the distribution of sample schools from which information was gathered and the types of data 
the schools provided. 


Exhibit 4. Cost Study Participant Sample of Kentucky Public High Schools Used for Analysis, by 
Locale and Free/Reduced Price Lunch Tercile in 2015-16 


City [ 1s) 


1 
Suburb 1p1,si, ss 1 
Town 11, si, ss 1 
Rural 15) 11, si, ss 11, si, ss 3 
Total 1 2 3 6 


Notes. Schools and districts had varying levels of participation in the three types of data collection. Subscripts 
indicate the type(s) of data collection to which schools submitted: DI = district interviews, SI = School Interviews, 
and SS = School Staff Surveys. Shaded cells indicated either nonparticipation or insufficient data. 


2.2.4 Assigning Prices 


The surveys and interviews were designed to capture the types and quantities of the resources used in 
the TI program. Once the resources were quantified, prices were assigned for each using a variety of 
sources based on the specific resource type and its characteristics. All prices were also adjusted (as 
appropriate) to represent their value in 2015 dollars. Below are details about how prices were derived 
for each ingredient type. 


The production of accurate cost estimates requires a comprehensive understanding of how each 
personnel position was compensated in its respective district; this includes both wages and benefits. The 
cost analysis team worked with KDE staff to create a database with district-specific, average full-time 
annual salaries and hourly wages for each certificated and classified position type identified during data 
collection, respectively. This information was entered into the RCM for each position type. 


In addition to an annual salary, all personnel are provided with benefits as a part of their employment 
compensation package. These benefits need to be included to represent the full dollar value of personnel 
time. To calculate benefits, the study team made use of the U.S. Census Annual Survey of Public School 
Finances (also known as the F-33 Form), which collects information regarding expenditures for each public 
school district in the country.?? The F-33 includes data on the total employee benefits and salary 
expenditures for every school district in the country by various position types (e.g. teachers, school 
administrators, instructional support staff, etc.). The study team calculated benefit ratios for each study 
district and school year based on job type by dividing total employee benefit expenditures by total 
expenditures on salaries. These ratios were multiplied by the district average salaries for each position to 
calculate the cost of benefits. The cost of benefits was then added to salaries to produce total 
compensation rates. County-level inflation rates were then used to inflate 2014—15 prices for personnel to 


13 For access to the F-33 data and corresponding documentation, visit https://nces.ed.gov/ccd/f33agency.asp. 
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2015 dollars.** Lastly, because all data were district specific, the final compensation rates had to be 
regionally adjusted to account for the fact that the cost of hiring and retaining staff varies across geographic 
labor markets. The study team used the Comparable Wage Index (CWI) developed by Dr. Lori Taylor of 
Texas A&M University to standardize these regional compensation rates to the state average.’ 


While average annual compensation rates for certificated job types were calculated for each study 
district, the reported staff time devoted to the different program activities was captured in hours. To 
this end, hourly compensation rates for certificated staff were needed to calculate the dollar value of 
time spent on the program. In turn, it was assumed that full-time administrators worked year round for 
a standard 40-hour work week, totaling 2,080 annual hours. Collective bargaining agreements and 
employee handbooks from the study districts were used to determine the number of hours worked by 
full-time teachers. Depending on the district, the annual hours worked that constituted a full-time 
teaching position ranged from 1,250 to 1,309 across the study sites. These figures were used to 
determine teachers’ hourly compensation rates at each site. 


This method of calculating compensation rates was applied to every staff position involved in the TI 
program, from superintendents to classified instructors. To better understand the calculations 
performed, consider a hypothetical district that paid teachers an average salary of $50,000 in 2014-15. 
If the benefit ratio for that position and district in this year was 40%, then the total compensation rate 
for the teacher would be $70,000. The prices from the 2014-15 school year were then multiplied by the 
local inflation rate to be reported in 2015 dollars. For example, if the inflation rate for this district was 
1.03, it would bring the compensation rate for teachers in the 2014-15 school year to $72,100 in 2015 
dollars. To adjust for regional personnel price differences, the compensation rate was then divided by 
the CWI. For example, if the CWI for the hypothetical district were 1.1 (i.e., it was 10% more costly on 
average to hire and retain staff), then the regionally and inflation-adjusted 2014—15 compensation rate 
for teachers in the district would be $65,545 a year in 2015-16 dollars (equal to $72,100 divided by 1.1). 
The overall salary was then divided by the number of hours a teacher works in the year (e.g., 1,250) to 
produce an hourly wage of $52.44 in regionally adjusted 2015 dollars. 


Prices for materials and equipment were collected in two ways. In some cases, staff shared exact prices 
in their survey and interview responses. For example, prices for professional development (e.g., 
“tuition”) and specific software subscriptions were explicitly reported by some study participants. In 
contrast, prices for materials and equipment were collected from national online retailers. Using the 
specifications described during data collection (e.g., functionality of the equipment, description of 
materials), the closest item match possible was found, and its respective unit price was entered into the 
RCM as part of the data collection process. Prices for materials and equipment were obtained in 2019 
for non-personnel items that were used in the 2014—15 and 2015-16 school years. This is important to 


14 The inflation factors used to convert staffing costs into 2015 dollars come from the U.S. Bureau of Labor Statistics 
Occupational Employment Statistics (OES) Metropolitan and Nonmetropolitan Area Occupational Employment and Wage 
Estimates for Education, Training and Library Occupations (Standard Occupation Code: 25000) for the specific regional areas of 
work in Kentucky in which our study sites were located (see https://www.bls.gov/oes/). 

15 For access to the CWI data and corresponding documentation, visit https://bush.tamu.edu/research/faculty/Taylor_ CWI/. 
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consider when assigning accurate prices to resources used in a program. Inflation rates were used to 
adjust the prices found in 2019 to 2015 dollars before including the costs in the main analysis.7° 


The prices of facilities used by the programs came from the CostOut Toolkit database developed by the 
Center for Benefit-Cost Studies in Education at Teachers College, Columbia University. The toolkit 
compiles national construction and maintenance information to provide a comprehensive database of 
facility prices. This source was used to identify the prices of classrooms and computer labs, the two 
types of facilities reported in our data collection. Price calculations used an average classroom size (750 
square feet, according to KDE) and a fully equipped computer lab design (as described in the Data 
Collection section) to assign prices for facilities. Like all other resources, facility prices were adjusted for 
inflation to reflect 2015 dollars. 


2.3 Calculating Costs 


Resource specifications were collected separately for the English language arts (ELA) and mathematics 
intervention services provided at each site. In addition, the RCM separated the specification and pricing 
information across resource and activity types. The costs of personnel time were calculated by 
multiplying the amount of time devoted to the various program activities and the hourly compensation 
rate for each respective position type. 


The costs of materials, equipment, and facilities required specialized calculations to ensure the 
appropriate value for each resource was achieved. The current cost analysis captures the annual value 
of several types of resources that are used over multiple years. Because these resources are not fully 
consumed in the year they are procured, it would be inappropriate to incur their full cost in the initial 
year in which they are acquired. Instead, the costs associated with materials and equipment were 
amortized across the respective product life spans to provide an annual cost for each year they are 
expected to be used.”” For example, the analysis assumed the useful life of a computer was 5 years, over 
which period its cost was amortized. Similarly, facility costs were amortized across a 50-year life span. 


The final step in the cost calculations was to put the total Tl program costs of ELA and mathematics 
intervention services at each school site into per-student terms. This was done by dividing the total 
intervention costs for ELA and mathematics, respectively, by the number of students who were enrolled 
in these types of interventions at each site.*® The calculated intervention costs are reported in the next 
section as a total cost per student of the Tl program, by resource type, and by activity type. In addition, 
costs are reported in the context of average per-student spending in the focus districts. 


16 Adjustments of the current (2019) prices for non-personnel resources to the 2 study years (2014-15 and 2015-16) were 
performed using the U.S. Bureau of Labor Statistics Consumer Price Index for Urban Consumers (series ID: CUUROOOOSAO), 
available for download from hitps://data.bls.gov/PDQWeb/cu. 

1’ The costs of all non-personnel resources were calculated using a 3.5% discount rate to account for depreciation of each asset 
over time. For more detail on calculating annualized costs of non-personnel resources see pages 89-90 from (Levin et al, 2018). 
18 The numbers of students taking ELA and mathematics interventions were measured using the counts who failed to meet the 
state benchmark thresholds on the ACT English/reading and mathematics assessments, respectively. 


Kentucky Targeted Interventions Program Cost Study—11 


2.4 Study Limitations 


The methods, data, and subsequent study findings provide prospective policy adopters and education 
practitioners with practical information about how the Kentucky TI program was implemented and the 
associated costs. However, to help put the findings in context, readers should be aware of some 
limitations of the study. The following section lays out these limitations and provides a lens through 
which future planned cost analysis of similar interventions can be viewed. 


The study team employed rigorous methods in an effort to ensure that results accurately represent 
intervention implementation and associated costs of the Tl program in the sample of participating 
districts/schools that varied with respect to incidence of FRPL and locale. However, the limited size of the 
sample, as well as the number of districts that participated and provided the required information for our 
analysis, greatly restricted our ability to generalize the findings across the state. Specifically, our analysis 
sample did not include schools from 6 of the 12 targeted locale/FRPL tercile stratum and therefore lacks 
representation.?? Districts and schools outside our analysis sample may have used a different mixture of 
intervention types than were captured in this study. In turn, the costs associated with those models are 
therefore also absent in the study results. An understanding of a wider variety of intervention models would 
allow for cost estimates that are more representative of what was implemented across the state. The small 
sample size also restricts our ability to make definitive claims regarding the relationship between 
implementation decisions driving costs (e.g., interventions delivered in a traditional classroom setting, 
computer-based learning, or a blended approach) and district characteristics (e.g., district wealth, historic 
student achievement, district size). 


Determining the incremental cost of implementing the TI program (i.e., the cost above and beyond what 
would have been spent in lieu of the program) also proved to be difficult. The Tl program was a statewide 
policy implemented in every school district in Kentucky with no districts, other than those in which all 
students met the ACT benchmark scores, continuing with their “business as usual” programming. Without 
comparison districts in the state, it is difficult to understand how TI program intervention services affected 
existing programming in the schools. For example, on the one hand, intervention services may have 
crowded out other educational programming that students otherwise would have taken. On the other, TI 
program services may have been introduced as an additional component to students’ existing educational 
programming. The presence of non-Tl program comparison districts and schools would have allowed for a 
better understanding of how the treatment and control groups differed. However, this particular policy 
which prescribed implementation in all districts/schools statewide made it impossible to observe business- 
as-usual schools alongside treatment schools during the same time frame. 


Future related research in other states would benefit from strategically phased implementation that 
allows for accurate district comparisons between schools receiving the treatment and comparison 
schools continuing with business-as-usual programming. Another potential solution for a lack of 
comparison districts would be to conduct a thorough cost analysis of the academic programming for 
students not receiving Tl program services and compare this to spending on TI program services. 
Unfortunately, extending this analysis to investigate the costs of non-Tl program services was outside 
the scope of this study. However, the calculated Tl-program costs presented in the next chapter still 


19 The representativeness of the sample was particularly lacking with respect to schools located in districts with relatively low 
incidences of FRPL (see Exhibit 4). 
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provide useful information. Specifically, they represent an upper bound of the program costs under the 
assumption that Tl services were provided on top of the existing educational programming. This would 
be the case if implementing the Tl program did not displace existing programming for any students 
(students who were referred to an intervention and those who met the ACT benchmark scores). 
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3. Results 


The results presented in this chapter include findings from both qualitative and quantitative analyses. 
The qualitative results describe how implementation of the Tl program varied across the study sites, 
which were given significant flexibility with respect to program design and delivery of intervention 
services. The quantitative findings illustrate the subsequent costs associated with the implementation 
decisions of each district. In turn, the qualitative results provide important context with which to 
interpret the quantitative findings. 


3.1 Qualitative Descriptions of Tl Program 


Implementation information was used to identify resources and calculate costs in a quantitative fashion. 
However, in-depth qualitative information collected as part of the interviews is used in this section to 
describe what occurred at the school level. The following provides an overview of the TI program as it 
was implemented in the study sites, based on information gathered from the interviews with school and 
district staff. It first describes common, general programmatic features of the TI program (i.e., initial 
planning meetings, referrals, monitoring, and professional development). This is followed by specific 
descriptions of the Tl program intervention models implemented across the sites. This includes how the 
intervention services were delivered (i.e., traditional classroom setting, computer-based learning, or a 
blended approach) and during what times services were provided (i.e., during the regular school day 
versus after school). The narrative highlights the programmatic variation across sites and builds context 
for the presentation of the quantitative cost results that follow. 


3.1.1 General Programmatic Features 

Implementation of the Tl program at each site began with initial planning leading up to the start of the 
school year. Districts also created referral procedures for students in need of intervention services along 
with protocols for monitoring student progress. In addition, various types of professional development 
to support implementation were used across the study sites. The following section provides descriptions 
of these general programmatic features. 


Initial Planning 

As is the case with any intervention, initial planning was necessary to design and implement the TI 
program. Schools and districts had varying approaches for the planning process, though there were 
several commonalities across sites. Teachers and administrators met as part of their professional 
learning communities (PLCs) to design the intervention at most study sites. The PLC structure was used 
for planning regardless of the mode of eventual intervention service delivery (i.e., off-the-shelf- 
curriculum, software package, teacher-designed curriculum, or a combination of modes). Guidance 
counselors and assistant principals were often involved in the initial planning process, while principal 
involvement was more supervisory in nature. 


Most school districts reported that they were available to support schools in the Tl program design 
process. However, the level of support provided by districts and desired by schools varied. Half of the 
participating districts described themselves as having an active role in the planning process, while the 
other half explained that most planning decisions were left to school discretion. 
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During the initial planning stages, most sites offered teachers professional development opportunities in 
preparation for implementation. For one site, this included attending a conference to learn about how 
to design and implement the TI program. For other sites, professional development focused on building 
the capacity of teachers and staff to effectively use newly purchased software packages intended to be a 
component of the TI program. Two schools reported securing a contract with a software company to 
facilitate face-to-face training sessions about specific software programs. 


Referring Students 

Each district had to define a referral process to identify and place students in the appropriate 
intervention(s). Most sites followed the recommended TI program regulations by using the state- 
sanctioned assessment (i.e., the ACT suite of assessments) as the sole indicator to refer students to the 
intervention program. However, one school in this sample leveraged its local autonomy to use end-of- 
course assessments as the TI program indicator instead of the ACT. Regardless of the mechanism for 
triggering a referral, schools held the primary responsibility of referring students who required 
additional supports and met the eligibility requirements set forth by the district. 


All schools in the sample followed district-specific guidelines for referring students to the TI program. 
However, there was variation across districts in how this process was implemented. The majority of 
districts reported working closely with school-level personnel to gather student data. This collaboration 
included content-area teachers, special educators, and guidance counselors. Data included ACT scores 
and subject specific performance data that guided the referral and intervention planning processes. 
District involvement also included discussions with school administrators about specific students using 
existing data and teacher familiarity with students to inform the referral process. In one site, school 
administrators created a team dedicated to processing the referrals for the entire senior class. This team 
was tasked with identifying eligible students and discussing relevant data on ELA and mathematics 
achievement to inform the subsequent intervention programmatic decisions for each student. 


Monitoring Student Progress 

Monitoring procedures often took the form of scheduled meetings among staff members, including 
teachers, administrators, and district-level personnel. Most districts left tracking and monitoring 
responsibilities to the school. However, in one school, the district staff met weekly with the school 
principal to discuss the intervention process and how students were progressing within the Tl program. 
School-based monitoring meetings varied in length and frequency. One high school reported that a 
3-hour team meeting was held each month to discuss student progress, while another met for 1 hour 
each week. Regularly scheduled teacher PLC meetings were also a vehicle for monitoring student 
progress and determining necessary program revisions to better support students. One site used 
summer working days and held daylong meetings among administrators to discuss student progress and 
make plans for the following year. 


School-based monitoring required a variety of resources, including teacher time and software programs. 
Although one school used Google Docs to track individual student programs and progress, the majority 
of sites relied on other software solutions. For example, schools that used software packages to deliver 
the bulk of the intervention services relied on those programs to track student progress. These software 
programs automatically collected performance data and, in many cases, provided a report of student 
progress. Infinite Campus, a product used to collect and organize student data and selected in 2006 by 
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KDE as the statewide student information system, was also widely used to document student progress 
and intervention service attendance. 


Professional Development 

The majority of sample sites reported school and/or district requirements to participate in professional 
development opportunities as a precursor to implementing the Tl program. These opportunities varied 
from longer, formal sessions to shorter and more informal trainings. For example, one site held formal 
professional development sessions during the summer prior to implementing the TI program. During 
these sessions, the superintendent met with principals and other administrators at the school to discuss 
how students would be identified, referred, and monitored throughout the TI program process. In 
another school, Tl program implementation included in-house informal training sessions for all teachers 
about how implementation would occur. In all sites, both formal and informal training opportunities 
were part of the planning process for the TI program. 


Professional development opportunities also gave teachers the information and support they needed to 
provide effective intervention services using the school-determined mode. As will be described later, 
some schools decided that additional software programs were the best way to meet students’ needs. In 
those instances, teachers required training on how to use and maximize the functionality of recently 
purchased software programs. This involved mandatory professional development events that focused 
on familiarizing teachers with the software functionality and the associated data analysis available to 
support student progress. Across the study schools, these types of training sessions were reported to 
take anywhere from a total of 1 to 18 hours. 


3.1.2 Descriptions of Interventions 

Information gathered through staff interviews and surveys shows that the types of intervention services 
varied across study sites, ranging from more traditional, teacher-led classroom instruction to computer- 
based activities to tutoring support. The day-to-day implementation of the TI program also took on 
varying forms across the sites. While some schools provided intervention services within existing 
required courses, others used an altered class schedule or after-school opportunities to serve students. 
In addition, the interviews and surveys captured information on various challenges faced implementing 
the TI program. The following section describes variation in implementation across the sites and 
associated challenges that were reported. 


Intervention Types 

Schools chose the types of interventions they felt best fit the needs of their students. These choices 
were also shaped by district and school capacity along with other logistical considerations (e.g., 
scheduling, resources). Some schools opted to embed the intervention programs into existing ELA or 
mathematics classes. This meant that students received intervention services within their required 
coursework and potentially alongside students who were not enrolled in the Tl program. Other schools 
designed stand-alone TI classes that focused on serving only students in need of intervention services, as 
identified through the referral process. In both the embedded and stand-alone course models, students 
were considered to have demonstrated mastery and fulfilled the requirement of the Tl program once 
they completed the assigned course. Many schools with the stand-alone courses also gave students the 
option of exiting the TI course before its conclusion by “testing out.” Most often, this meant that a 
student had to score above a certain benchmark on an assessment taken through the Kentucky Online 
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Testing (KYOTE) service. Another option was for a student to pass an assessment that was included in 
the school’s purchased software package used to deliver the intervention services. 


Computer-based software programs were used as a mode of service delivery for the Tl program 
implemented in five of the study schools. The rationale provided by schools for using these programs 
was that they allowed for individualized tailoring of instructional support. These programs were used for 
both classroom activities and homework. The software programs were also used to track progress, 
analyze performance data, and administer assessments to check for content mastery and inform 
instructional decisions. Most schools used educational software for at least a portion of their Tl program 
offerings, and some schools used software packages as the main component of their intervention 
programs. In these cases, the intervention required students to work on individual computers. Site staff 
explained that students often worked through the software program material in a school computer lab 
with the guidance and support of a staff member. The aim of the software programs was to provide 
each student with individualized educational experiences. In addition, software programs were reported 
to be adaptive in nature and responded to student performance with appropriately challenging material. 
Participants explained that the functionality of these types of programs enabled teachers to meet the 
needs of students with differing abilities while also preparing students for standardized test-taking 
platforms. Software packages were used for both ELA and mathematics interventions and included 
Reading Plus, DreamBox, IXL, Study Island, Plato, Edgenuity, and ALEKS. 


Implementation of Tl Services 

The type of staff allocated to deliver intervention services was determined by the programmatic structure 
in each school. This often included a full time-equivalent, certified teacher for each core subject content 
area (e.g., English and mathematics). These teachers focused on providing intervention services for 
students enrolled in the Tl program, but may have performed other non-program duties as well. 
Depending on the service delivery structure in each school, these dedicated teachers either were the 
primary instructor for content-specific courses or oversaw and supported the use of computer software 
programs. Teachers also spent time in PLCs or other collaborative environments to develop the curriculum 
and make necessary programmatic and instructional changes to meet students’ evolving needs. 


Although the allocation of full-time, certified teachers was the most common service delivery structure 
across all sampled sites, one school stood out in its instructional staffing choices for the Tl program. This 
school relied predominately on classified (hourly) staff to provide instruction for students in the Tl 
program. Classified staff are generally less expensive to employ based on the terms of employment and 
required qualifications. The primary mode of intervention service delivery for this particular school 
consisted of an off-the-shelf software solution and curriculum programs, with a classified staff member 
facilitating sessions. In addition, this school provided all the intervention services within the confines of 
the school computer lab. Aside from this single outlier, all schools utilized full-time, certified instructors 
to deliver instructional services in a variety of modes. 


Logistical management of program implementation varied across schools as a function of school 
structure, staff capacity, and student needs. One site reported that a majority of the senior class fell 
short of the required benchmarks in a single year. As a result, nearly every student in that senior class 
qualified for some type of Tl-related intervention delivered either during the regular school day or as 
after-school programming. This school created a unique service delivery model to meet the needs of the 
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struggling seniors, as well as a majority of the overall school population. Each month, teachers identified 
struggling students and grouped them based on skill strengths and deficits. Students were then excused 
from their typical classes for most of a single day, enabling them to attend approximately 5 hours of TI 
intervention-focused instruction and coursework. This deviation from the typical school day occurred 
several times during the year for all students who required intervention services. 


An additional component of this school’s TI program structure was an after-school tutoring program 
open to all students. Although the tutoring program was not required, students who were eligible for 
the Tl program were strongly encouraged to attend. Each week of the school year, teachers offered 
specialized classes designed for small-group instruction. The after-school sessions focused on building 
specific skills that teachers felt students needed. In this unique school setting, all interventions—both 
those offered during regular school hours and those offered during after-school programs—were 
designed by certified teachers. Software packages were only occasionally used to enhance teaching, 
based on students’ needs and teacher instructional style. 


Challenges 


At the start of implementation, many sites faced difficulties integrating the Tl program into the pre- 
existing school culture. In some cases, interviewees described difficulty generating buy-in from teachers 
and other staff members, as well as from students and families. These sites had trouble making teachers 
feel they had an adequate voice in the curriculum design process and other decisions. There were also 
occasional concerns about interventions relying too heavily on software programs for instruction. 
However, these challenges were not insurmountable. Two schools reported that they persuaded initially 
reluctant teachers, parents, and students by engaging stakeholders in revising the intervention process. 
Over the course of the first year, this type of collaboration built acceptance and support for the 
program. Staff at one school explained that after the conclusion of the Tl program mandate, they 
revisited their model and made important adjustments. While the school kept its current software, they 
decided to de-emphasize the software’s prominence in the Tl program and instead enable teachers to 
integrate it into their own intervention curricula as they saw fit. Through that process, the school was 
able to re-engage teachers delivering intervention services. 


With newly required educational programs, there are often concerns about adequate resources to meet 
requirements while continuing with existing initiatives. In our study sample, staff at two sites voiced 
concerns related specifically to the Tl program and their available budget. Participants at these sites 
explained that they did not feel their budgets were sufficient to purchase adequate materials and 
allocate personnel to the TI program. They felt these costs could not be absorbed into their existing 
budget structure. As a result, some of these schools decided to embed the TI program services into 
existing courses. Other schools experiencing similar concerns felt forced to cut some existing classes to 
free up resources for stand-alone TI program courses. Aside from budgetary constraints, schools 
reported concerns about the logistical burden of the program. Specifically, some schools found it 
challenging to fit the Tl program into the existing high school student schedules. At times, there were 
competing priorities for students to fit coursework and intervention programming into the school day. 


Implementation information is important to understand for educational policies, especially those that 
allow for local decision making. like the Tl program. The information provided in this section gives 
policymakers an understanding of how the study sites implemented the TI program along with 
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challenges they faced. While there likely was more implementation variation across the state, this 
sample illustrates how some districts and schools actualized the Tl program policy required by the state. 
This information also provides context for the variation in the quantitative results described in the next 
section, in which the calculated costs of Tl program intervention services are reported. These calculated 
intervention costs are described in a variety of ways. First, we provide the overall costs per student of 
ELA and mathematics interventions for each school in the cost study sample and as an average across all 
the sites. Next, we present the average intervention costs per student across the school sites by 
resource type (i.e., personnel, materials/equipment, facilities, tuition). Finally, we report average 
intervention costs per student by activity category (administration, instruction, monitoring, professional 
development, and referrals). Lastly, we present the calculated intervention costs for each site in the 
context of overall spending. 


3.2 Costs of the Tl Program 


This section describes the results from the quantitative cost analysis. First, the overall per-student costs 
of the ELA and mathematics interventions offered under the TI program are provided for each study 
school and as averages across the study sample. Next, these costs are broken out by resource type (i.e., 
facilities, materials/equipment, personnel, and tuition) and activity (i.e., administration, instruction, 
monitoring, professional development, and referral), respectively. Finally, the costs of the TI program 
are put into the context of the overall spending across the study sites. 


3.2.1 Overall Costs 


Our analysis draws upon the data on costs derived from the quantities and prices captured through data 
collection and organized in the RCM. In our analysis, we calculated costs for each individual resource 
reported by respondents as being used in the implementation of their respective ELA and mathematics 
interventions in the 2014-15 and 2015-16 school years. Information was then aggregated within the 
sites to arrive at the overall school-specific costs of the TI program for each of the 2 study years. Next, 
the costs for the first study year (2014-15) were adjusted for inflation to represent 2015 dollars. All 
costs were then regionally adjusted to account for differences in the cost of hiring and retaining staff 
across the different regional labor markets in the state in which sample sites are located. Finally, the 
overall costs of the ELA and mathematics interventions for each school and study year were put into 
per-student terms by dividing by the total number of students who did not make the benchmark cutoff 
scores on the relevant ACT assessment (and therefore would have been referred to a targeted 
intervention). 


Exhibit 5 presents the average per-student costs of the ELA and mathematics interventions that were 
implemented across the six study schools over the 2-year study period.”° The average per-student cost 
of the ELA interventions implemented across the school sample over the 2-year study period was $569, 
while the corresponding cost of the mathematics interventions was $613. 


20 Specifically, the figures represent the pupil-weighted average costs per student of the ELA and mathematics interventions 
across the study schools. 
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Exhibit 5. Average per-Student Costs of ELA and Mathematics Interventions Implemented 
Under the Kentucky TI Program Across Study Schools Over the Study Period (in 2015 Dollars) 
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The findings show a broad range of costs associated with interventions across the sample sites. 

Exhibits 6 and 7 provide an overview of how the average per-student costs of ELA and mathematics 
interventions delivered under the Tl program over the study period varied across the study schools. As 
the exhibits illustrate, the average cost per student participating in an ELA intervention ranged from 
$123 to $1,228, while for mathematics the range was $110 to $1,076. The yellow columns represent the 
per-student average costs across the study schools previously presented in Exhibit 5. 


As described in the preceding section, the types of interventions and the manner in which they were 
implemented varied across sites. This variation may account for the wide range of calculated costs 
across schools. However, other factors also may have driven the cost differences, such as the numbers 
of students who received TI program services (listed under the site labels in the exhibits that follow), 
which ranged on average across the school sites over the 2-year study period from 85 to 274 for ELA and 
from 95 to 277 for mathematics. This diseconomies-of-scale explanation (i.e., the fact that 
districts/schools operating at a smaller scale tend to have higher per-student costs due to a smaller 
number of students over which costs associated with fixed inputs can be spread) may help explain the 
high costs per student calculated for Site A, which served the smallest number of students of all the 
study schools. However, we would expect to also see upward pressure on the per-student costs for 
Site E, which served the second smallest number of students. Instead, Site E shows the lowest per- 
student costs. This latter result can be explained by the staffing decisions made in Site E, whereby far 
less expensive, classified (hourly) staff were used exclusively to deliver instruction. 
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Exhibit 6. Average per-Student Costs of ELA Interventions Implemented Under the Kentucky 
TI Program Across Study Schools Over the Study Years (in 2015 Dollars) 
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Exhibit 7. Average per-Student Costs of Mathematics Interventions Implemented Under the 
Kentucky TI Program Across Study Schools Over the Study Years (in 2015 Dollars) 
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In order to provide additional context to the average per-student costs presented above the study team 
developed indicators denoting when the intervention took place (timing) and how it was delivered 
(structure) using the reported descriptions of the interventions offered at each sample site. Specifically, 
timing indicates whether the intervention took place within versus outside the regular school day, while 
structure is categorized as computer-based, teacher-led or blended (a combination of computer-based 
and teacher-led). Exhibit 8 provides the structure and timing of the ELA and mathematics interventions 
implemented at each study school. 


Virtually all of the schools delivered interventions in both subject areas in class periods within the 
regular school day. However, school C also reported delivering ELA and mathematics intervention 
services after school. Because there were no sites that delivered interventions solely outside of the 
regular school day it is impossible to draw associations between costs and implementation timing. There 
was more variation with respect to intervention structure. Specifically, four schools implemented 
computer-based interventions for ELA (schools A, B, D and E) and mathematics (schools A, B, E and F). 
Two schools implemented teacher-led interventions for ELA (schools C and F), with one of these also 
using a teacher-led intervention for math (school C). Finally, one school chose a blended approach for 
their mathematics intervention (school D). 


Exhibit 8. Structure and Timing of ELA and Mathematics Interventions Implemented at Study 
Schools 


Computer-Based A, B, D, E A, B, E, F 
Teacher-Led C, F C C C 
Blended D 


Exhibits 9 and 10 present the average per-student costs of the ELA and mathematics interventions 
implemented across the sample schools by intervention structure. As might be expected, across the 
sample schools the average cost per student of delivering both computer-based interventions for ELA 
and mathematics was less than for those that were teacher led by $160 and $224, respectively. In 
addition, the average per-student cost of delivering computer-based mathematics interventions was 
also less than for blended learning. However, we note that these results should be interpreted with 
caution due to the small numbers of schools upon which they are based and the fact that there are a 
variety of factors including implementation decisions and school characteristics potentially driving the 
costs that may not be associated with intervention structure. 
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Exhibit 9. Average per-Student Costs of ELA Interventions Implemented Under the Kentucky 
TI Program Over the Study Years by Intervention Structure (in 2015 Dollars) 
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Exhibit 10. Average per-Student Costs of Mathematics Interventions Implemented Under the 
Kentucky TI Program Over the Study Years by Intervention Structure (in 2015 Dollars) 
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3.2.2 Costs by Resource Type 


The data on resource costs were also organized by type across common categories of resources 
(personnel, materials/equipment, facilities, and tuition). Personnel resources were clearly a key driver of 
the overall costs of providing intervention services. Exhibit 11 shows the average breakdown of per- 
student cost for ELA Tl program interventions by resource type across the school sites over the study 
period. On average, $532 per student, or 93% of the overall ELA intervention cost, went to personnel. 
The remaining $37 per student, on average, was spent on materials/equipment ($29), facilities (S5), and 
tuition ($3). Exhibit 12 shows the corresponding figures for mathematics interventions, which follow a 
similar pattern: $576 (94%) of the average per-student costs were devoted to personnel, while $38 per 
student was spent on the other resource types, with materials and equipment accounting for most ($31 
or 5%) of this remaining cost. These findings align well with the intention of the program. Specifically, Tl 
that meets the needs of individual students requires significant personnel effort for their development 
and implementation, which includes targeted student referrals, direct instruction, and subsequent 
monitoring of results. 


Exhibit 11. Average per-Student Costs of ELA Interventions Implemented Under the Kentucky 
TI Program Across Study Schools Over the Study Period, by Resource Type (in 2015 Dollars) 
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Exhibit 12. Average per-Student Costs of Mathematics Interventions Implemented Under the 
Kentucky TI Program Across Study Schools Over the Study Period, by Resource Type (in 2015 
Dollars) 
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Exhibits 13 and 14 provide average cost breakouts by resource type of the ELA and mathematics 
interventions, respectively, over the study period on a school-by-school basis. As shown in the average 
cost figures presented above, a majority of the overall intervention costs were attributed to personnel. 
In five of the six study schools, personnel accounted for at least 91% of the cost of both the ELA and 
mathematics interventions, while personnel for the remaining school (Site E) accounted for 59% of these 
intervention costs. 
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Exhibit 13. Per-Student Costs of ELA Interventions Implemented Under the Kentucky TI 
Program Across Study Schools Over the Study Years, by Resource Type (in 2015 Dollars) 
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Exhibit 14. Per-Student Costs of Mathematics Interventions Implemented Under the Kentucky 
TI Program Across Study Schools Over the Study Years, by Resource Type (in 2015 Dollars) 
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Note. The overall mathematics intervention cost per student is shown in boldface. Labels for dollar values less than 
$20 are not shown. 


While we have seen that non-personnel costs made up a relatively small portion of the overall cost 
across all schools, it is important to note that they represent critical resource requirements for 
implementing the various interventions. The most universally used non-personnel resource was 
materials/equipment (e.g., textbooks, software licenses, etc.), which was reported by all study schools. 
Usage of facilities outside of the regular classroom (e.g., computer labs) and tuition for professional 
development were reported by two school sites (Sites D and E for facilities and Sites A and D for tuition). 
However, these two resource types often represented much smaller shares of intervention costs. In 
sum, for both ELA and mathematics interventions, the corresponding costs were predominantly 
attributable to personnel, although some non-personnel resources were reported, mostly in the form of 
materials/supplies. 


3.2.3 Costs by Activity 


The policy behind the TI program preserved local control by giving districts and schools the autonomy to 
select their intervention models and implement them with maximum latitude. While this flexibility 
enabled sites to tailor interventions to best meet the unique needs of their students, it posed a 
challenge to the systematic collection of resource data organized according to the ways in which 
resources are used. However, as data were collected and analyzed, several activity patterns began to 
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appear. The study team used these patterns to develop a common set of activities observed across all 
school sites. Specific tasks were then categorized into activity types to offer a more detailed account of 
resource usage and the associated costs. The activity categories discovered during data collection, as 
described in Exhibit 2, were implementation activities related to administration, instruction, monitoring, 
professional development, and referral. Thus, our analysis offers an additional lens through which to 
view costs that may be of interest to policymakers and practitioners. Gaining a better understanding of 
costs by activity category can inform future policy decisions that aim to provide an optimal balance of 
resource autonomy and accountability. 


Exhibit 15 shows how ELA implementation costs were distributed, on average, across the five activity 
categories. As expected, the majority of costs associated with ELA interventions are attributable to 
instruction, which accounted for $466, or 82% of the overall per-student cost. This was followed by $42 
per student, or 7% of the overall cost devoted to identifying and referring students to interventions, and 
S40, or 7% on monitoring student progress. The remainder of the costs were split between professional 
development activities ($15, 3%) and administration ($7, 1%). Exhibit 16 depicts the distribution of 
average mathematics intervention costs across the five activity categories. While we observed across 
activities a pattern of average resource allocation similar to that which emerged for ELA interventions, 
the cost share is slightly more heavily weighted toward instruction, which is offset by slightly smaller 
shares of cost devoted to monitoring, professional development, and referrals. 


Exhibit 15. Average per-Student Costs of ELA Interventions Implemented Under the Kentucky TI 
Program Across Study Schools Over the Study Period, by Activity (in 2015 Dollars) 


$7, 1% 


$15, 3% 
5 
= Administration 
= Instruction 


= Monitoring 
= Professional Development 


= Referral 


Kentucky Targeted Interventions Program Cost Study—28 


Exhibit 16. Average per-Student Costs of Mathematics Interventions Implemented Under the 
Kentucky TI Program Across Study Schools Over the Study Period, by Activity (in 2015 Dollars) 
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Exhibits 17 and 18 report the site-specific breakouts of ELA and mathematics intervention costs by 
activity, respectively. The variation in the patterns of cost by activity across the study schools is notable. 
For ELA, a majority of the cost for all but one site was attributable to instruction (Exhibit 15). Site F 
differed from other schools in this respect for two reasons. First, this site reported a significant amount 
of involvement on the part of school- and district-level staff both in referring students to interventions 
and in monitoring their progress. Second, Site F implemented its ELA intervention differently than other 
sites. Rather than offering a stand-alone class section or after-school session (strategies used by other 
sites), the site devoted about 20% of a regular English class period to ELA intervention instruction, which 
resulted a relatively small share of the intervention cost being attributable to instruction. 


Three of the other sites (Sites A, C, and D) had more moderate costs for monitoring and referral, while very 
little of the intervention cost at two sites (Sites B and E) was attributable to these noninstructional activities. 
Fairly similar patterns of cost by activity category emerged for the mathematics interventions (Exhibit 16). 
Because Site F treated its mathematics intervention as a stand-alone course, the school spent significantly 
more on instruction for the mathematics intervention than on instruction for its ELA intervention. 
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Exhibit 17. Per-Student Costs of ELA Interventions Implemented Under the Kentucky TI 
Program Across Study Schools Over the Study Years, by Resource Activity (in 2015 Dollars) 
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Exhibit 18. Per-Student Costs of Mathematics Interventions Implemented Under the Kentucky 
TI Program Across Study Schools Over the Study Years, by Resource Activity (in 2015 Dollars) 
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Note. Overall mathematics intervention cost per student is shown in boldface. Labels for dollar values less than 
$20 are not shown. 


3.2.4 TI Costs in the Context of Overall Spending 


It is helpful to view the calculated costs of the Tl program interventions in the context of overall current 
expenditures. Average current per-student spending expressed in 2015 dollars for all students (both 
those receiving and not receiving Tl program services) over the 2-year study period (2014-15 and 2015-— 
16) ranged from $6,845 to $11,063 across the study schools, with $9,291 spent on the average student 
in the study school sample.7? 


The study team calculated the average spending per intervention student and non-intervention student 
by subtracting the calculated ELA and mathematics intervention costs from the current spending figures 
for each of the study schools to show how much, on average, was spent on (a) students who were not 
participating in an intervention and (b) students who did participate in an ELA and/or a mathematics 


21 Data on current spending per pupil were taken from the publicly available data posted on the KDE website. Specifically, we 
used the Learning Environment datasets called “Students/Teachers” for the 2014—15 and 2015-16 school years located here: 
https://applications.education.ky.gov/SRC/DataSets.aspx. The definition of current spending can be found in the KDE data 
glossary here: https://applications.education.ky.gov/SRC/Glossary.aspx. The 2014-15 per-pupil spending figures were 
transformed into 2015 dollars using rates based on the OES wage data (see footnote 10, above) and then averaged with the 
2015-16 figures for each site. 
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intervention. The latter Tl program spending was broken out by that devoted to instruction versus 
noninstructional activities. 


Exhibit 19 shows the resulting average spending across the study schools. The findings suggest that, on 
average, $9,552 was spent on a student participating in an ELA and/or a mathematics intervention 
under the TI program. Of this figure, $8,515 (89%) was spent on providing non-intervention services to 
the typical 12th-grade student enrolled in the six study schools. Spending on the average student 
participating in one or more (ELA and/or mathematics) interventions was $1,037 (11%), which included 
$852 (9%) to cover instruction costs and $185 (2%) for costs associated with the other four activity 
categories (administration, monitoring, professional development, and referrals). 


In sum, the results suggest that the cost of the Tl program over the study period accounted for about 
11% of the overall current spending on the average student, with the bulk of this cost going toward 
instruction. However, it would be incorrect to infer that the cost of delivering TI program services was in 
addition to spending that existed before implementation of the program. Rather, referrals to Tl program 
instruction likely replaced other programming (courses or other intervention services) that students 
otherwise would have taken. 


Exhibit 19. Average per-Student Spending on Program Intervention and Non-intervention 
Services Across Study Schools Over the Study Period (in 2015 Dollars) 
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Exhibit 20 shows per-student spending on non-intervention services assumed to be spread across all 
students and those services associated with the ELA and mathematics interventions provided to 
students under the TI program. Overall per-student spending on those taking an ELA and/or 
mathematics intervention ranged from $7,234 to $11,254 across the study schools. The amount spent 
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on providing intervention services ranged from $193 to $1,972 per student (2% to 19% of overall per- 
student spending), a majority of which was attributed to instruction costs. 


Exhibit 20. Per-Student Spending on Program Intervention and Non-intervention Services 
Across Study Schools Over the Study Period (in 2015 Dollars) 
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4. Conclusion 


Kentucky’s Tl program mandated that all students be evaluated in Grades 8, 10, and 11 using the ACT suite 
of assessments. If students did not demonstrate college and career readiness using these measures, they 
were referred to the Tl program to receive appropriate supports. Districts and schools had flexibility in how 
they provided and monitored those interventions. Some chose to incorporate intervention programming 
into existing courses, while others opted to provide additional programming to meet students’ needs. 


The cost analysis presented in this study serves as a companion to AIR’s TI program impact study. The goal 
of this cost analysis was to capture the resources used to implement the program for students receiving 
intervention referrals in the 12th grade to aid KDE staff and district- and school-level Tl program developers, 
as well as other stakeholders, in developing a better understanding of the policy and its associated costs. To 
this end, we used the ingredients approach to produce comprehensive estimates of the cost of the 
individual programs implemented by schools by resource type and activity for the 2-year study period 
(school years 2014—15 and 2015-16). This approach enabled the study team to separate time spent by staff 
on the TI program from their other duties and to further break out intervention-related activities. 


The study found that the average per-student cost of Tl program ELA interventions across the study 
schools was $569; the corresponding average per-student cost of TI program mathematics interventions 
was $613. However, there was substantial variation in the TI program per-student costs across the 
individual study schools. Specifically, the per-student cost of the ELA interventions implemented as part 
of the TI program ranged from $123 to $1,228 across the study schools. The corresponding figures for 
the TI program mathematics interventions ranged from $110 to $1,076. It should be noted that across 
the study schools, many students participated in both types of interventions. 


As the study team expected, the costs associated with personnel accounted for the largest portion of 
intervention costs. On average, personnel accounted for 93% of the ELA intervention costs and 94% of 
the mathematics intervention costs. On average, instruction composed 82% of the overall costs of the 
ELA interventions and 85% of the overall mathematics intervention costs. Putting the Tl program costs in 
the context of spending, the findings suggest that the amount spent on the average 12th-grade student 
receiving an ELA and/or mathematics intervention under the TI program across the study schools was 
$9,552, of which $1,037 (11%) could be attributed to intervention services. However, intervention 
spending on these students varied widely across the study sites, ranging from $193 to $1,972 per 
student (equal to 2% and 19% of overall spending, respectively). 


Future research activities may include pairing the cost calculations with estimates of the program 
impacts on various student outcomes generated by the companion study (Backes, Oliveira, & Goldhaber, 
2020) to better understand the extent to which the benefits of implementation outweigh its costs. For 
example, the findings in the impact study measure the degree to which the intervention was effective in 
reducing the probability that a student enrolled in a college program requires a remedial course in 
English or mathematics. These results could be used in concert with the TI program cost calculation 
estimates to better understand whether the cost savings associated with ameliorating the need for 
students to take remedial courses outweigh investments in the TI program. 
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Appendix A. Tl Program Online School Survey 


Questions Response Details 
Involvement in Targeted Intervention 


1) Please select the school year(s) in which you were involved in administering the 
Targeted Interventions Program or delivering instructional interventions to 12th 
graders under the program at your school. 


2) During the 2014-15 or 2015-16 school years, were you responsible for working 
directly with students to deliver instructional interventions to 12th graders under 
the Targeted Interventions Program? 


3) During the 2014-15 or 2015-16 school years, were you responsible for any 
administrative duties under the Targeted Interventions Program? 


4) During the 2014-15 or 2015-16 school years, were you involved in any 
professional development required to perform administrative or instructional duties 
under the Targeted Interventions Program? 


Involvement in Targeted Intervention 


5) Check the box that best describes your primary job title in the 2014-15 school 
year. 


6) Check the box that best describes your primary job title in the 2015-16 school 
year. 


7) Including the 2014-15 and 2015-16 school years, how many years did youserve | e Total years 
in the job title(s) specified? e Years in this district 
e Years in this school 


8) What percentage of full-time did you work at this school in the 2014-15 and e Percentage of full-time worked 
2015-16 school years? 


9) How many instructional/non-instructional days per year and hours per day under | e Instruction days per year 
contract were you required to work in the 2014—15 and 2015-16 school years? e Hours per instructional day 

e Non-instruction days per year 

e Hours per non-instructional day 


10) In the typical week, how many total hours did you work onsite and off-site at e Onsite Hours 
your school over the 2014-15 and 2015-16 school years? e Offsite Hours 


Instructional Activities 


11) Were you involved in English/Reading, Math, or both interventions? e English/Reading 
e Math 
e Both subjects 
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Questions 


12) During the typical week in the 2014-15 and 2015-16 school years how many 


hours did you spend on each of the following activities delivering English/Reading 


Targeted Interventions to 12th-grade students? 


13) How many weeks did the typical English/Reading Targeted Intervention you 
delivered in the 2014—15 and 2015-16 school years last? 


14) How many students were enrolled in the typical English/Reading Targeted 
Intervention session delivered during or outside the regular school day in the 2014— 
15 and 2015-16 years? 


15) During the typical week in the 2014-15 and 2015-16 school years how many 
hours did you spend on each of the following activities delivering Math Targeted 
Interventions to 12th-grade students? 


16) How many weeks did the typical Math Targeted Intervention you delivered in 
the 2014-15 and 2015-16 school years last? 


Response Details 


Direct contact with students 
delivering intervention 
instruction during the regular 
instructional day 

Direct contact with students 
delivering intervention 
instruction outside the regular 
instructional day (before or 
after school) 

Grading, preparing, and 
planning for intervention 
instructional sessions 

Meeting with intervention 
students outside of regularly 
scheduled instructional sessions 
Interacting with parents of 
intervention students 


Lengths in weeks 


During regular school day 
Outside regular school day 


Direct contact with students 
delivering intervention 
instruction during the regular 
instructional day 

Direct contact with students 
delivering intervention 
instruction outside the regular 
instructional day (before or 
after school) 

Grading, preparing, and 
planning for intervention 
instructional sessions 

Meeting with intervention 
students outside of regularly 
scheduled instructional sessions 
Interacting with parents of 
intervention students 


Length in weeks 
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Questions Response Details 


17) How many students were enrolled in the typical Math Targeted Intervention e During regular school day 
session delivered during or outside the regular school day in the 2014-15 and 2015— | e Outside regular school day 
16 years? 


Non-Personnel Resources 


18) How many different English/Reading Targeted Interventions did you deliver in e Number of interventions 
the 2014-15 and 2015-16 school years? 


19) How many English/Reading Targeted Interventions did you deliver in each type |e Classroom 

of location in the 2014—15 and 2015-16 school years? e Library 
e Auditorium 
e Room in the district office 
e Multipurpose room 


20) For the typical English/Reading Targeted Intervention you delivered in the 2014—| e Art supplies 
15 and 2015-16 school years, please use an “X” to indicate which materials and e Computer 
equipment were used by students. e Internet connection 
e Office Supplies 
e Textbooks 
e Workbooks 


21) For the typical English/Reading Targeted Intervention you delivered in the 2014— | e Computer 


15 and 2015-16 school years, please use an “X” to indicate which materials and e Internet connection 
equipment you used. e Interactive display 
e Office supplies 
e Overhead projector 
e Smartboard 
e Textbook (teacher edition) 
e Workbook (teacher edition) 


22) How many different Math Targeted Interventions did you deliver in the 2014-15 | e Number of interventions 
and 2015-16 school years? 


23) How many Math Targeted Interventions did you deliver in each type of location |e Classroom 

in the 2014-15 and 2015-16 school years? e Library 
e Auditorium 
e Room in the district office 
e Multipurpose room 


24) For the typical Math Targeted Intervention you delivered in the 2014-15 and e Art supplies 
2015-16 school years please us an “X” to indicate which materials and equipment e Computer 
were used by students. e Internet connection 
e Office Supplies 
e Textbooks 
e Workbooks 
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Questions 


25) For the typical Math Targeted Intervention you delivered in the 2014-15 and 
2015-16 school years please us an “X” to indicate which materials and equipment 
you used. 


Professional Development 


26) How many total hours did you spend on the following professional development 
or collaboration activities required for your Targeted Interventions Program duties 
in the 2014-15 and 2015-16 school years? 


27) What were the total approximate costs of the formal training workshops, 


courses, and conferences in which you participated in the 2014-15 and 2015-16 
school years? 


Administrative Activities 


Response Details 


Computer 

Internet connection 
Interactive display 

Office supplies 

Overhead projector 
Smartboard 

Textbook (teacher edition) 
Workbook (teacher edition) 


Attended formal training 
workshops offered within 
district 

Attended formal training 
workshops outside of district or 
college courses 

Attended conferences 
Received formal or informal 
coaching or mentoring from 
another teacher or staff 
member 

Collaborated with other 
teachers to review individual 
student work 


Formal training workshops or 
college courses 
Conferences 
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Questions Response Details 


28) How many total hours did you spend on the following administrative activities Providing training to staff on 

for the Targeted Interventions Program at your school in the 2014—15 and 2015-16 the Targeted Intervention 

school years? Program 
Reviewing ACT data to identify 
students requiring interventions 
Determining school demand for 
different types of targeted 
interventions 
Working with instructors to 
plan and adopt targeted 
interventions 
Scheduling targeted 
interventions 
Selecting and referring students 
to targeted interventions 
Supervising delivery of targeted 
intervention instruction 
Monitoring student compliance 
with attending assigned 
targeted interventions 
Collecting, analyzing, and/or 
reporting data on student 
completion of targeted 
interventions 


Personal Information 


29) What is your gender? 
30) Which best describes you? (Please check all that apply) 
31) What is the highest level of schooling you have completed? 


Compensation 


32) What was your approximate annual gross salary (before taxes and deductions) 
in 2014-15 or 2015-16? 


33) If relevant, please list the amounts of the following types of additional 


compensation you received for performing administrative and/or instructional 
duties under the Targeted Interventions Program in the 2014-15 and 2015-16 
school years. 
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Appendix B. TI Program School and School 
District Interview Protocol 


Questions 


Program Administration 


1. [SCHOOL DISTRICT AND SCHOOL] | would first like to ask you about how the Targeted Intervention Program 
was administered in your district in the study years (2014-15 and 2015-16). 
a. Which staff were primarily in charge of covering the administration of the program? 
b. Please describe how students were referred to different interventions. 
i. Was this primarily a district- or school-level function or shared between the central district 
office and schools? 
What data were used and where did these come from? 
iii. Which district-level staff were involved and approximately how much time did they spend 
coordinating student referrals to different interventions? 
c. [SCHOOL DISTRICT] Please describe the monitoring of student attendance and completion of the 
interventions to which they are referred. 
i. Was this primarily a district- or school-level function or shared between the central district 
office and schools? 
Which district-level staff were involved and approximately how much time did they spend 
monitoring student attendance and completion? 
How much staff time was dedicated to compiling and reporting data on intervention 
attendance and completion to the district and/or state? 


c. [SCHOOL] Please describe the monitoring of student attendance and completion of the interventions to 
which they are referred. 


i. Which staff were involved and approximately how much time did they spend monitoring 
student attendance and completion? 
How much staff time was dedicated to compiling and reporting data on intervention 
attendance and completion to the district and/or state? 


d. [SCHOOL] Were there meetings surrounding the initial implementation and regular operation of the 
Targeted Intervention Program at your school? 
i. Ifso, which staff members were involved and how much time did they spend preparing and 
participating in these meetings? 


[SCHOOL DISTRICT AND SCHOOL] Did your school/district invest in any computer hardware or software 
solutions to implement the Targeted Intervention Program? 

Probe for: Software installation process, training for administrators and/or teachers to use the 
hardware/software, etc. 


[SCHOOL DISTRICT AND SCHOOL] Were there any other investments in non-personnel (materials, supplies or 
equipment) associated with the implementation of the Targeted Intervention Program? 


[SCHOOL DISTRICT AND SCHOOL] Please describe the administration of the ACT exams? Which district-level 
staff were involved in coordinating the ACT assessment process and how much time did they spend? 
Probe for: Scheduling exam, administering exam, data collection and reporting. 


Intervention Delivery Questions 


Kentucky Targeted Interventions Program Cost Study—41 


Questions 


[SCHOOL DISTRICT AND SCHOOL] Can you please provide a general description of interventions implemented 
for 12th graders at [high school here] during 2014—15 and 2015-16 school years as part of the Targeted 
Intervention Program? 

Probe for: Please list the interventions that were provided for Reading/Writing and Mathematics. 


a. Howdid your district or the school decide on which interventions would be offered each year? 
Probe for: Were there internal meetings to discuss this? If so, how many meetings were there, how 
long were they, and who from the district was in attendance? 


For each intervention listed ask the following in order: 


a. Was the intervention offered in one or both of the study years (2014-15 and 2015-16)? 
If both: Were there differences in how the intervention was delivered between the years? 


Was the intervention branded (off-the-shelf) or developed by your district or school? 
Probe for: Was the intervention delivered by school or district staff, or by staff external to the school 
and district (i.e., privately contracted)? 


Was the intervention delivered as part of an existing class, as a stand-alone (whole period) course, or 
through some other arrangement? 


When and where did the intervention take place? 

Probe for: Within regular school day, before/after school, during summer. Probe for: Classroom, 
library, auditorium, other space. What was the room size/capacity and what would it be used for 
instead of the intervention? 


Was the intervention delivered by school/district staff or contracted out to a third party? 
Listen for, but don’t necessarily probe: Types of personnel that were involved in delivering the 


intervention (teachers, instructional support, etc.) and how much time these staff spent in delivering 
the intervention. Did the Targeted Intervention Program effect scheduling and staffing for regular 
school programming and operations? 


Did the district office require any professional development for staff (administrative or instructional) 
for the particular intervention before or during the study years (2014-15 and 2015-16)? 

Probe for: Types of professional development required to provide the intervention and corresponding 
time invested by district/school staff or contractors. 


Were there any investments in non-personnel (materials, supplies or equipment) associated with the 
targeted intervention? 

Probe for: Types of non-personnel required to provide intervention services and corresponding time 

invested by district/school staff or contractors. Were the non-personnel resources provided by your 

school or by your district? 


Miscellaneous Questions 


[SCHOOL DISTRICT AND SCHOOL] Did the district or school require any more general (i.e., not intervention- 
specific) professional development for staff (administrative or instructional) involved with the Targeted 
Intervention Program before or during the study years (2014-15 and 2015-16)? 


[SCHOOL DISTRICT AND SCHOOL] Besides the costs we’ve just talked about, what other district-level personnel 
or non-personnel resources were required to deliver the Targeted Intervention Program courses? 


[SCHOOL DISTRICT AND SCHOOL] What challenges did the district face implementing the Targeted Intervention 
Program and how were these addressed? 
Probe for: How much time was spent on addressing the challenges and who was involved? 
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Appendix C. Study Overview Used for 
District/School Recruitment 


mM 


AMERICAN INSTITUTES FOR RESEARCH" 


Overview: 


Our 
Children, 


Kentucky Targeted Interventions Cost Study 


Targeted Interventions (TI) are an important component of Kentucky’s comprehensive system of education reform 
and represent one of the key levers to fulfilling the Commonwealth’s commitment to college- and career-readiness 
for all students. In 2014-15 and 2015-16, the program required students in Kentucky’s public education system 
who were deemed to need additional support in reading, writing and/or mathematics to be referred to targeted 
instructional interventions. The American Institutes for Research (AIR) has partnered with the Kentucky 
Department of Education to conduct the TI Cost Study to investigate the time, effort, and subsequent costs involved 
in providing instructional services at Kentucky high schools under the TI program. 


STUDY PURPOSE 

The study will enhance an existing statewide research 
evaluation of the effects of TI services on student 
college readiness outcomes being conducted by AIR by 
adding a cost analysis of the effort involved in 
implementing these services. Specifically, the Tl Cost 
Study will identify the personnel (e.g. time spent by 
staff) and non-personnel used to deliver TI services in a 
selected number of school sites and translate these 
specified resources into dollar values. The findings of 
this study will be used in concert with those of the 
outcome evaluation findings to develop measures of 
cost-effectiveness (i.e., the cost per-unit of outcome 
generated by the services delivered under Tl). 


Specific study questions include: 

1. What types of targeted interventions were 
provided to 12" graders across Kentucky high schools 
in the study years? 

2. What personnel and non-personnel resources 
were devoted to implementing targeted interventions 
and what were their costs? 

3. How cost-effective are targeted interventions 
as a means of improving college and career readiness? 


DATA COLLECTION AND ANALYSIS 

AIR has randomly selected 12 districts and one high school in 
each district to serve as the study sample. Participation in the 
study is entirely voluntary. The study will involve the 
administration of a 30-minute interview with one staff member 
at each sample district and high school, respectively. In 
addition, an online survey taking no longer than 30 minutes will 
be administered to certificated school staff at each sample high 
school. The information to be gathered will identify personnel 
time and non-personnel resources used to implement TI 
services at each study site. These data will be used in 
conjunction with average salary rates and market prices of non- 
personnel items to calculate the costs associated with providing 
TI services. 


EFFORTS TO MINIMIZE BURDEN 

The study team has extensive experience in conducting 
interviews and administering surveys to district- and school- 
level certificated staff and will make every effort to minimize 
burden to all staff providing information for the study. As 
mentioned above, the length of both the district and school 
staff interviews will be 30 minutes. Also, the burden 
associated with the school staff survey is expected to be 
minimal, given most certificated school-level staff will not 
have been involved in delivering TI services and therefore will 
be exited from the survey after an initial filter question. For 
those staff who have been involved in the delivery of TI 
services the expected survey will be a maximum of 30 
minutes. 
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TIMELINE 


The target date for completion of the data collection 
is by the end of February 2019. A final report for the 
study is expected in mid-2019. 


STUDY LEADERSHIP 


The study will be conducted by the American 
Institutes for Research, a well-established and 
reputable behavioral and social science research 
organization. The cost study research team will be 
led by Principal Research Economist, Dr. Jesse Levin, 
Ph.D. who is collaborating with Chief Performance 
Officer, Karen Dodd of the Kentucky Department of 
Education Office of the Commissioner. 


CONTACT INFORMATION 


Jesse Levin, Ph.D. 

Principal Research Economist 
American Institutes for Research 
(650) 376-6270 


jlevin@air.org 
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